`

download all files that they serve? Let’s use bash to carry such a

task:

#!/bin/bash

FILE="${1}"

OUTPUT_FOLDER="${2}"

if [[ ! -s "${FILE}" ]]; then 1

echo "You must provide a non-empty hosts file as an argument."

exit 1

fi

if [[ -z "${OUTPUT_FOLDER}" ]]; then

OUTPUT_FOLDER="data" 5

fi

while read -r line; do

url=$(echo "${line}" | xargs) 2

if [[ -n "${url}" ]]; then

echo "Testing ${url} for Directory indexing..."

if curl -L -s "${url}" | grep -q -e "Index of /" -e "[PARENTDIR]"; then 3

echo -e "\t -!- Found Directory Indexing page at ${url}"

echo -e "\t -!- Beginning a recursive download to the \"${OUTPUT_FOLDER}\" folder..."

mkdir -p "${OUTPUT_FOLDER}"

wget -q -r -np -R "index.html*" "${url}" -P "${OUTPUT_FOLDER}" 4

fi

fi

done < <(cat "${FILE}")

You can download this script from

https://github.com/dolevf/Black-Hat-

Bash/blob/master/ch05/directory_indexing_scanner.sh.

In this script, we define the FILE and OUTPUT_FOLDER

variables. Their assigned values are taken from the arguments the

user passes on the command line ($1 and $2). We then fail and exit

the script (exit 1) if the FILE variable is not of the file type and

not of length zero (-s) 1. If the file has a length of non-zero, it

means that the file isn’t empty (and some data was written into it).

We then use a while loop to read the file at the path assigned to

the FILE variable. At 2, we ensure that each whitespace character in

each line from the file is removed by piping it to the xargs

command. At 3, we use curl to make an HTTP GET request and

follow any HTTP redirects (using -L). We silence cURL’s verbose

output (using -s) with and pipe it to grep to find any instances of

the strings Index of / and [PARENTDIR]. These two strings

Black Hat Bash (Early Access) © 2023 by Dolev Farhi and Nick Aleks